Conventionally, as image sensors having an image processing function by integrating an analog processing circuit with an image sensor serving as a solid-state image sensing element, for example, semiconductor chip structures (Japanese Patent Laid-Open Nos. 2000-13694 and 11-266002) and the like are known, in which a solid-state image sensing element and a spatial filter processing function and the like are integrated.
Examples of attempts to add the early visual function of a living body to an image sensor are a study of silicon retina by Mead et al. (C. Mead, “Analog VLSI and Neural Systems”, Addison Wesley Pub., 1989, U.S. Pat. No. 4,786,81) and an artificial retina LSI which integrates a pixel array, pixel access control circuit, multiplexer, and weight control circuit on one chip to realize a predetermined filtering function (Journal of the Institute of Image Information and Television Engineers Vol. 53, No. 2, pp. 178–183, 1999).
The operation of a CMOS sensor circuit will be described below as a prior art. FIG. 8 is a schematic circuit diagram of a sensor circuit (a drive circuit diagram of a CMOS sensor) disclosed in Japanese Patent Laid-Open No. 11-196332. This sensor circuit outputs a video signal to, e.g., the spatial filter circuit of the above-described image sensing signal processing circuit.
Referring to FIG. 8, photoelectric conversion elements are formed by photodiodes PD11 to PD22 . . . with p-n junctions and transfer MOS transistors ST11 to ST22 . . . connected to the cathodes of the photodiodes. A vertical selection circuit VSR sequentially sets outputs V1 to V8 to “H” to sequentially activate vertical selection lines HL1, HL2, . . . . In addition, outputs H1 to H8 from a horizontal selection circuit HSR are sequentially set to “H” to turn on horizontal transfer switch MOS transistors HT1 to HT8. Image charges accumulated in the respective pixels are time-serially read out from vertical output lines HV1 to HV8 to an output line HOL and output through an amplifier AMP.
A reset MOS transistor RES is turned on every time pixel charges are output, thereby resetting the output line HOL every time pixel charges are output. Of the photodiodes PD11 to PD22 . . . , the photodiodes PD11, PD12, PD21, PD22, PD31, . . . which are connected to the vertical output lines HV1 and HV2 are shielded from light on their cathode sides to extract dark charges.
FIG. 9 is a timing chart of the interlaced read of the solid-state image sensing apparatus. The pixels are read out every other pixel in the horizontal and vertical directions of photoelectric conversion elements in the effective signal range.
First, the vertical selection circuit VSR sets the output V1 to “H” to set the vertical selection line HL1 to “H”. Subsequently, the outputs H1, H2, H3, H5, H7, . . . from the horizontal selection circuit HSR are sequentially set to “H”. With this operation, the horizontal transfer switch MOS transistors HT1, HT2, HT3, HT5, HT7, . . . are sequentially turned on. Image charges accumulated in the respective pixels are time-serially read out from vertical output lines HV1, HV2, HV3, HV5, HV7, . . . to the output line HOL and output through the amplifier AMP.
After the outputs H1, H2, H3, H5, H7, . . . from the horizontal selection circuit HSR are sequentially set to “H”, the reset MOS transistor RES is turned on to reset the output line HOL every time the pixel charges are read out, thereby eliminating the influence of adjacent pixels.
Referring to FIG. 9, an output OUT from the amplifier AMP changes to “H” first because the reference signal output photoelectric conversion elements of the photodiodes PD11 and PD12 of the pixels are shielded from light. After that, a low-potential-side level is output in accordance with the charges of the first effective signal output photoelectric conversion elements of the photodiodes PD13, PD15, and PD17. In addition to such an interlaced read, a block read for reading an arbitrary range from an entire image is also done.
The interval of the interfaced read can be arbitrarily changed. A prior art which applies a spatial filter after the above-described read is disclosed in Japanese Patent Laid-Open No. 2000-13694.
However, in the above prior art, a plurality of predetermined different geometrical features cannot be efficiently sequentially extracted from time-serially input image data by the single circuit.
In addition, to extract a plurality of predetermined different geometrical features using an array circuit in which feature extraction circuits are spatially parallelly arrayed, time-serial image signals from a sensor must be spatially separated and individually input to corresponding feature extraction circuits. Furthermore, a sensor output must have one-to-one correspondence with the address of a feature extraction circuit on the array circuit.
Image recognition or voice recognition schemes are roughly classified into a type for sequentially calculating and executing a recognition processing algorithm as computer software that is specialized to a specific recognition object and a type for executing recognition processing by a dedicated parallel image processor (e.g., an SIMD or MIMD machine).
Typical examples of image recognition algorithms for calculating a feature vector associated with the similarity to a model to be recognized are a method using similarity calculation or higher-order correlation coefficient calculation, which is done by representing the model data of an object to be recognized as a template model and executing template matching with an input image (or its feature vector), and a method using hierarchical parallel processing (Japanese Patent No. 2741793).
Especially, to fully detect even a partially occluded object to be detected, a scheme of determining the similarity to a local portion of a model to be detected is proposed in Japanese Patent Laid-Open No. 11-15945. In this proposal, matching processing between a local model and a local portion of an object is executed, and the degree of support for supporting the presence of the object at each local portion is obtained by integration processing based on the Dempster-Shafer theory or Fuzzy theory, thereby doing highly accurate detection.
In Japanese Patent Laid-Open No. 6-176158, a plurality of types of feature vectors for an input pattern are individually recognized by obtaining a single category similarity. Then, the results are normalized by the maximum similarity value to obtain the integrated similarity, thereby executing final recognition.
In Japanese Patent Laid-Open No. 9-153021, in a parallel processing apparatus which sequentially receives digital signals and integrates one or more processing results obtained by divisionally processing the digital signals by one or more processors, the digital signals are divided using a means for converting an input signal into unit signals, thereby obtaining an optimum arrangement in terms of capability or economics in accordance with the input signal.
However, of the above prior arts, the arrangement disclosed in Japanese Patent Laid-Open No. 11-15945 does not explain how to integrate matching results when object models have a plurality of categories, and which local model yields matching results to be integrated. In this prior art, confidence degrees for the presence of features by a non-additive measure are integrated on the basis of, e.g., the Dempster-Shafer theory, though it does not guarantee optimum estimation.
In addition, if the size of an object in an image is different from that of an object model, or if a plurality of objects with different sizes simultaneously exist, the method is hard to apply. Even if object models corresponding to a plurality of sizes are prepared in advance, and similarity to each model is determined sequentially for the difference sizes, it is not preferable from the viewpoint of processing efficiency or circuit scale (memory size) In the parallel processing apparatus disclosed in Japanese Patent Laid-Open No. 9-153021, appropriate data division processing can hardly be executed when a plurality of objects having different sizes are present in image data or the like. Hence, when the type or size of object is unknown, appropriate pattern recognition can hardly be executed by parallel processing only by simply uniquely dividing data in synchronism with an input signal.
The arrangement of the pattern recognition apparatus disclosed in Japanese Patent Laid-Open No. 6-176158 cannot improve the memory efficiency or reduce the circuit scale. Generally, in an arrangement for executing pattern recognition processing using hierarchical parallel processing circuits (e.g., the arrangement disclosed in Japanese Patent No. 2741793), a plurality of features are simultaneously parallelly detected at sampling points on input data. In any of the above prior arts, hence, the number of elements in the lower layer increases depending on the size of an input image, resulting in an increase in circuit scale.
Conventionally, as a logic IC that allows a user to construct an arbitrary logic, an FPGA (Field Programmable Gate Array) is provided.
An FPGA is formed by periodically arraying a plurality of relatively large circuit blocks and interconnection blocks on a chip. A number of “devices capable of programming” electrical connection or disconnection of the circuits are arranged in the circuit blocks and interconnection blocks. A user can design connections in and between the blocks in a field (site of use) by programming (defining) these devices. Cell library blocks having various kinds of logic circuits are arrayed in a matrix in a chip. Signal interconnections are formed horizontally and vertically between the cell library blocks. Switch elements are arranged at the interunits of the interconnections and between the interconnections.
Three schemes below are employed as switch elements.
(1) Amorphous Si is inserted between metal interconnections to insulate them in the initial state. A current is supplied between desired metal interconnections to break the insulating film, thereby short-circuiting the interconnections (anti-fuse scheme). Conversely, metal interconnections are short-circuited in advance using polysilicon or the like and then disconnected by a laser or the like (fuse scheme).
(2) A normal MOS transistor is used as a switching element, and the potential state of its gate is stored in a memory. As the memory, a volatile DRAM or SRAM, or a nonvolatile EEPROM is used. Use of a ferroelectric memory (FRAM) is also proposed. A volatile memory must receive the information of logic connection and wiring connection into the FPGA every time it is used. However, this memory can be a Reconfigurable Logic capable of changing information a number of times. To the contrary, a nonvolatile memory can maintain the information of logic connection and wiring connection even when the power supply is turned on/off.
(3) A switching element itself is formed from an EEPROM or the like to store the ON/OFF information of a transistor.
In designing a logic system using an FPGA, a desired logic function is described by an HDL (Hardware Description Language) using a support tool provided by an EDA vendor, and logic circuit data (the type of logic gate to be used and inter-terminal connection information) is obtained using a logic synthesizing tool. To implement the resultant logic circuit, the logic operation data of a basic logic cell CLB and connection data between the basic logic cells CLB are obtained. In addition, interconnections to be used and the ON/OFF control data of a cross-point switch CSW are determined from the connection data, thereby building a whole logic circuit system. Such logic design can easily be done by the user, and its convenience is widely recognized.
Arrangements disclosed in Japanese Patent Laid-Open Nos. 11-168185 and 2000-331113 are known as methods of reconfigurably connecting an analog processing element using an FPGA. In the former arrangement, an FPGA is formed on one surface of a stacked substrate, an analog processing circuit is formed on the other surface, and input and output terminals and an interface circuit, which connect the FPGA and analog processing circuit, are prepared. In the latter arrangement, first and second analog signals are converted into PWM (Pulse Width Modulation) signals and input to an FPGA circuit, and logical operation of the two PWM signals is executed by the FPGA circuit, thereby allowing reconfiguration.
A hierarchical neural network disclosed in Japanese Patent No. 2679730 is related to the architecture of a hierarchical neural network that allows time-divisionally multiplexed use of single-layered hardware, and has as its object to allow equivalently making a multi-layered structure by time-divisionally using single-layered hardware. This prior art discloses a hierarchical neural network formed by connecting a plurality of neuron models to each other, comprising a single-layered unit assembling means for forming a single-layered unit assembly by installing a plurality of units of neuron models which allow to time-divisionally output a voltage through a nonlinear output function by generating a product of a time-divisionally multiplexed analog signal and external digital weight data and by time-divisionally adding the product through a capacitor to integrate the product, a feedback means for feeding back the output from the single-layered unit assembling means to the input unit of the same single-unit assembly, and a control means for executing control to time-divisionally multiplex analog signals from units, which are output from the single-layered unit assembling means, and to time-divisionally use the single-layered unit assembling means through the feedback means, wherein the single-layered unit assembling means is time-divisionally used to equivalently form a neural network having a hierarchical structure.
In an FPAA (Field Programmable Analog Array) circuit according to U.S. Pat. No. 5,959,871, programmable analog processing cells including a multiplexer, demultiplexer, control circuit, and analog processing element are parallelly arrayed through signal lines, thereby executing programmable analog processing.
In all the above prior arts, a so-called FPGA is based on a logic circuit. That is, only the connection state between logic blocks can be changed. Hence, only with the FPGA, analog parallel arithmetic processing of a neural circuit network or like cannot be implemented.
In applying the arrangement of Japanese Patent Laid-Open No. 11-168185 to a neural circuit network, when the number of neural elements increases, the number of input/output terminals exponentially increases, resulting in difficulty in handling. For this reason, in principle, it is difficult to set an arbitrary interconnection structure, and a special process for mounting the elements on a stacked substrate is required. In the arrangement disclosed in Japanese Patent Laid-Open No. 2000-331113, it is basically possible to form a pseudo addition circuit or pseudo integration circuit for two analog signals. However, it is difficult to implement a large-scale parallel hierarchical processing circuit.
In the hierarchical neural network disclosed in Japanese Patent No. 2679730, since no means for variably arbitrarily controlling the inter-layer connection is prepared, the number of types of processing that can be substantially realized is considerably limited.
In the arrangement according to U.S. Pat. No. 5,959,871, each of the analog processing units (cells) has a control circuit, analog processing circuit element, signal branch circuit, and the like. For this reason, the scale of the circuit component (analog processing cell) becomes very large, and therefore, the total circuit area inevitably increases. Additionally, in a structure in which analog processing units each incorporating a multiplexer and demultiplexer are arrayed on a two-dimensional plane, it is difficult to realize an arbitrary connection structure and, more particularly, hierarchical connection.